The present invention relates to a method of converting a multicolor image in which each pixel only has a single one of the colors into a full color image. The invention further relates to apparatus employing the method.
Due to the trichromatic nature of the human visual system, each of the pixels of xe2x80x9cfull colorxe2x80x9d digital images are associated with at least three independent information items, that is the intensities of three independent colors (e.g. red, green, and blue).
Digital cameras and video devices which create such images use an array of light sensors to collect information about an image scene, and each sensor corresponds to a respective pixel. However, to reduce cost, many such digital cameras employ a color filter array (CFA) in which each light sensor is covered by a single type of filter, and is therefore sensitive to only one of the three colors.
For example, FIG. 1 shows a 5xc3x975 pixel section from a well-known CFA called the Bayer pattern (see U.S. Pat. No. 3,971,065 to B. E. Bayer, the disclosure of which is incorporated by reference). In FIG. 1, the pixels are numbered 1 to 25, and each pixel is marked with the corresponding color intensity value produced by the camera. The color intensity value consists of a letter (R, G or B) indicating the color to which the corresponding sensor is sensitive, and a subscript indicating the enumerated number of the pixel. Thus, for example, G5 is the green color intensity of pixel five, at the top right of FIG. 1. Because the human visual system is more sensitive to green patterns than blue or red ones, there are twice as many green filters as red or blue ones.
Since each pixel in the image produced by a CFA is associated with a single color intensity value, the image is not a full-color image. However, it can be used to generate a full color image by what are called xe2x80x9ccolor interpolationxe2x80x9d (or xe2x80x9cdemosaicxe2x80x9d) algorithms. In other words, the image produced by the camera has at each pixel an intensity value for one of the three colors, and the intensity values of the two other colors (the two xe2x80x9cmissing intensity valuesxe2x80x9d) are estimated by the color interpolation algorithms.
For example, in one such algorithm (the xe2x80x9cnearest neighborxe2x80x9d algorithm) any missing intensity value at a given pixel is estimated to be equal to the intensity value of the same color at the closest neighboring pixel which contains an intensity value for that color. A more sophisticated variant of this algorithm is the xe2x80x9cbilinear interpolationxe2x80x9d algorithm in which any missing intensity value at a given pixel is estimated to be the mean value of the intensity values of the same colors of the closest neighboring pixels which contain an intensity value for that color. Both these algorithms are discussed for example in an article by J. E. Adams, xe2x80x9cInteractions between color plane interpolation and other image processing functions in electronic photography,xe2x80x9d Proceedings of SPIE, vol. 2416, pp 144-151, the disclosure of which is incorporated herein by reference in its entirety.
One of the key difficulties of such algorithms is that in the resulting image the hues of adjacent pixels can change abruptly and in an unnatural manner. To address this the xe2x80x9csmooth hue transition interpolationxe2x80x9d algorithm (also discussed in the article by J. E Adams mentioned above) uses bilinear interpolation to interpolate the green pixels (enough pixels have a green intensity value for this to produce good interpolation), and interpolates the red and blue intensity values at each pixel using the derived green intensity value at that pixel. Specifically, the blue intensity value at each pixel is interpolated from the green intensity value at that pixel and the blue intensity values of the nearest neighbor pixels having a blue intensity value. Interpolation of the red values is performed analogously.
The algorithms above perform averaging among neighboring pixels indiscriminately. This causes the xe2x80x9czipperxe2x80x9d artifact in which a color edge is interpolated in a way which appears as a fault to a human visual system. To address this problem authors such as Sakamoto T. et al (xe2x80x9cSoftware pixel interpolation for digital still camera suitable for a 32-bit MCUxe2x80x9d, IEEE Trans. Consumer Electronics, pp.1342-1352, Nov.1998) proposed an xe2x80x9cedge sensingxe2x80x9d algorithm in which the edges of objects in the image are estimated by noting abrupt changes in the green color intensity value, indicative of edges in objects, and attempting to interpolate the missing color intensity values of each pixel by an average only over neighboring pixels of the same object.
Other interpolations include: the color correction algorithm (see for example U.S. Pat. No. 5,629,734 to Hamilton Jr. et al.), which employs the 2nd Laplacian derivative along possible edge directions; the variable number gradients algorithm (Chang E, Shiufun Cheung, Pan D, xe2x80x9cColor filter array recovery using a threshold-based variable number of gradients,xe2x80x9d Proc. SPIE, vol.3650,1999, pp.36-43) which computes 1st order Laplacian derivatives of missing colors as gradients in a localised 5xc3x975 window centered at each pixel under consideration; and xe2x80x9cpattern recognition algorithmsxe2x80x9d which attempt to recognise patterns within the data (see for example U.S. Pat. No. 4,630,307, to D. R. Cok).
The present invention seeks to provide new and useful methods for interpolating color intensity values, and additionally apparatus employing the methods.
The invention is motivated by the observation that there is a high degree of color correlation between different color channels in natural scenes. A simple model for color images is one of Lambertian non flat surface patches. The color signal received by the camera is the reflected signal of the surface patch from a light source with direction vector l, and it is reasonable to assume that the reflective surface is homogenous, so that the received signal at spatial location (x, y) can be written as:
IR(x, y)=cR(x, y)N(x, y)xc2x7lxe2x80x83xe2x80x83(1) 
IG(x, y)=cG(x, y)N(x, y)xc2x7lxe2x80x83xe2x80x83(2) 
IB(x, y)=cB(x, y)N(x, y)xc2x7lxe2x80x83xe2x80x83(3) 
where N(x, y) is the surface normal, and ci(x, y) (for i equal to one of R, G or B) is the albedo capturing the characteristics of the reflective materials. Thus the dot product N(x, y)xc2x7l is the reflected light level. If the reflective surface is homogenous in a localised region ci(x, y)=ci, therefore:                                                                         I                i                            ⁡                              (                                  x                  ,                  y                                )                                                                    I                j                            ⁡                              (                                  x                  ,                  y                                )                                              =                                                                      c                  i                                ⁡                                  (                                      x                    ,                    y                                    )                                                                              c                  j                                ⁡                                  (                                      x                    ,                    y                                    )                                                      =                                          c                i                                            c                j                                                    ,                            (        4        )            
which is constant. That is, the color ratio within a given object in a localised region is constant. The present inventors have found experimentally that the above model is consistent with image data collected from natural scenes.
In general terms the present invention proposes that for a pixel for which an intensity value of a first color is known, the intensity value of a second (missing) color is interpolated as the known intensity value of the first color multiplied by the local average of the second color and divided by the local average of the first color. Each of the intensity values of the first and second color values is determined over a set of nearest neighboring pixels.
For example, motivated by eqn. (4) we propose that for a pixel k having a known red color intensity value Rk, the missing local green intensity value Gk is interpolated as:                               G          k                =                                            G              _                        ⁢                          xe2x80x83                        ⁢                          R              k                                            R            _                                              (        5        )            
where {overscore (G)} and {overscore (R)} are the average green color and red color in the localised region. The value {overscore (G)}/{overscore (B)} thus constitutes a color correction factor, relating the known red color intensity value Rk and the missing local green intensity value Gk.
Similarly, Bk can be estimated by replacing Gk with Bk and {overscore (G)} with {overscore (R)}. Of course, in pixels k in which is known, Gk takes the place of Rk in equation (5) and {overscore (G)} takes the place of {overscore (R)}, and they used to derive Rk and Bk. Similarly in pixels k in which Bk is known, Bk takes the place of Rk in equation (5) and {overscore (B)} takes the place of {overscore (R)}, and is used to derive Rk and Gk.
Each of the averages {overscore (B)}, {overscore (G)} and {overscore (R)} is calculated over a respective set of neighboring pixels for which the corresponding color intensity value is a known value (i.e. not a missing color intensity value). It is preferable that these sets of pixels are small (e.g. only 4 pixels each), to minimise unnecessary color smoothing. The respective sets of pixels for calculating {overscore (B)}, {overscore (G)} and {overscore (R)} span respective areas (here called xe2x80x9cwindowsxe2x80x9d; we may for example define each window precisely as the convex hull of the centres of the pixels in the set) in the image which in general overlap. Preferably, in the case of set s of 4 pixels, the windows are square. In the case of the color having a known color intensity value in the pixel under consideration, the set of pixels used to find the local average of the intensity value of this color preferably includes the known pixel itself.
To find all the missing color values for a given pixel, a number of windows is formed which is equal to the total number of colors in the image. Preferably, the proportional overlap of these windows should be high, to improve the accuracy of the color correction factor, which is just the weighted average ratio of the color intensities for different color channels of the overlapping windows.
Averaging the color intensities results in a smoothing artifact since if there is an edge in the windows the average is supported by pixels to either side of the edge. Having said that, this factor applies to both the averages over {overscore (G)} and {overscore (R)}, so these effects largely cancel each other when their ratio is taken. Since eqn (5) also contains Rk, the interpolated value of Gk should be highly sensitive to which side of the edge the k-th pixel lies in.
Thus, in general terms, in the present invention the known color intensity values give an accurate positioning of the edges, and these edges support the interpolation of the missing color values. For this reason the smoothing artifact produced by the present invention is much less than, for example, in the bilinear interpolation algorithm described above. On the other hand, the computational complexity of the present algorithm is about the same.
Specifically, one expression of the invention is a method for processing a digitized image having pixels aligned in rows and columns and having color intensity values for each of a plurality of colors, each pixel having a color intensity value for only one of the colors, the method including for successive first pixels:
deriving a first local average value, which is the local average value of the color intensity value of the color which is present in the first pixel;
deriving a respective local average value of each of the other colors;
for each of the other colors deriving a respective color correction factor proportional to the local average of that other color and inversely proportional to the first local average value; and
deriving an interpolated color intensity value for each of the other colors as the color intensity value of the color which is present in the first pixel multiplied by the respective color correction factor.
Naturally, this definition includes any mathematically equivalent series of operations.
The local average values of each color are determined using, for each color, a set of pixels in the neighborhood of the first pixel and having a color intensity value for that color. In the case of the color for which the first pixel has a color intensity value, the set of pixels for that color may include the first pixel itself. Each of the local average values is then found as an average over the color intensity values of the corresponding set of pixels.
Note that this xe2x80x9caveragexe2x80x9d may simply be the arithmetic mean of the set of color intensity values of the set. However, it is not limited in this respect, and may for example be a different average, such as the median. Alternatively, the xe2x80x9caveragexe2x80x9d may give a different weight to the color intensity values of different ones of the set of pixels.